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ABSTRACT 



The evaluation of systemic educational reforms entails a 
paradigm shift among evaluators who must change the lens through which they 
look at the reforms. This paper proposes methodological and design 
alternatives to measure nontraditional outcomes, such as changes in policy, 
over the lifespan of the reforms. Systemic reforms include intended and 
unintended outcomes and the actual outcomes that result . Outcomes of 
educational reform can also be organized in terms of whether they result from 
top-down or bottom-up approaches. A research approach to measuring reform 
outcomes will be more methodologically soxind if the variables of interest are 
measured in multiple ways. By comparing findings using multiple qualitative 
and quantitative data collection strategies, evaluators can identify trends 
and lead reformers to midcourse corrections when needed. By defining outcomes 
and measuring those outcomes, evaluators and reformers will be driving the 
reforms. (Contains 17 references.) (SLD) 
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Overview 



One of the main reasons for looking at systemic educational reform from an 
evaluation perspective is to determine what happens as a result of these 
interventions In this very high stakes process, different players within the reform 
as well as outside of it are interested in finding out if their investments in terms of 
time as well as financial and human resources are really producing measurable 
change. However, since systemic educational reform takes place in an 
everchanging environment, the evaluator must look beyond casual relationships 
and traditional definitions and methodologies when examining the outcomes of 
such a program. 

An evaluator of systemic educational reforms should address what happens to 
and within the unit of change targeted by the reform; this unit of change could be 
defined as the teacher, the classroom, the school or the university to name a few 
possible options. Each one of these units of change will require a different level 
of intervention to collect data and a different level of analysis to reach valid 
conclusions that will be useful to participants as well as to stakeholders. Each 
one of these units of change will require a different evaluation design particularly 
as the reforms by their own systemic nature require the study of the changes that 
occur within all major elements of the system being transformed as well as within 
some of the other systems that are interrelated with the target system as part of 
their evaluation. Therefore, the evaluator’s use of multiple approaches and 
methodologies is highly advisable (Datta, 1997). 

The evaluation of systemic educational reforms entails a paradigm shift among 
evaluators who must change the lens which they use to look at these reforms. 
Examples of non-traditional outcomes such as changes in educational policy, 
leveraging of resources formation of partnerships, increases in parent 
involvement, level of participant empowerment are a few of the variables of 
interest to evaluators of systemic educational reforms that will be discussed here. 
Methodological and design alternatives to measure these variables over the 
lifespan of the reforms through innovations such as coordinated series of studies 
will be proposed in this paper (see Caracelli & Greene, 1997). 

Definition of Outcomes and Outcome Variables 

Weiss (1998) describes outcomes as “the end results of the program for the 
people it was intended to serve” (p.8) and further elaborates that outcomes are 
interchangeable with results and effects. Outcomes are certainly an end result of 
systemic educational reforms as well as of many other types of programs, but the 
nature and context of these initiatives requires a wider definition. For example, in 
systemic educational reforms, outcomes can be evident at the level of the 
classroom, school, district, or state. Evaluators of systemic educational reforms 
are usually interested in connections between different levels of interventions and 
outcomes as well as in factors that contributed to the occurrence of those 
outcomes. 
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Because of the additional dimensions of systemic educational reforms that 
differentiate these programs from other educational interventions, distinctions 
between outcome variables and outcomes need to be established. In systemic 
educational reform, an outcome variable is a quantity, dimension, or quality of the 
system subject to change because of the initiative. A systemic variable is an 
outcome variable that can be measured across the system such as student 
academic achievement in Science and Mathematics. In turn, an outcome for a 
systemic initiative is a change in an outcome variable directly attributable or likely 
attributable to the initiative such as improvements in student learning as a result 
of participation in standards-based instruction in Science and Mathematics. 

Outcomes can be organized into three different types: (1) intended, unintended, 
and actual; (2) main effects (i.e., changes in one variable); and (3) mixed effects 
(i.e., changes in the interaction among two or more variables). The following 
section introduces the reader to this classification. 

Types of Outcomes in Systemic Educational Reform 

In the design of any systemic reform, the initiative usually stipulates a series of 
outcomes that it wishes to achieve within a specific time period given its goals 
and objectives. Three examples of possible intended outcomes from Science 
and Mathematics educational reforms are: (1) showing measurable increases in 
student academic achievement in Science and Mathematics; (2) having a 
predetermined number of schools fully implementing a standards-based reform; 
and (3) increasing communication between management and governance 
structures. However, regardless of how well a systemic reform plans its 
interventions and sets out its goals and objectives, it may not achieve all of its 
intended outcomes or, instead, it may attain other unintended outcomes which 
may become windows of opportunity to refocus the reform and its evaluation. 
Whether they were intended or unintended, the evaluator of systemic reform 
works with the actual outcomes of the reform being studied. 

Measuring main effect outcomes entails that the evaluator establishes a clear 
connection between the reform and a particular outcome variable. In contrast, 
mixed effects outcomes requires that the evaluator understands the different 
factors that are contributing to the presence of particular outcome variables. 
Given the dynamic and complex nature of systemic reforms and of the context 
where they are being implemented, most evaluators spend most of their time 
working with mixed effects outcomes. However, they will very likely find that the 
same outcomes can be considered main effects in some initiatives and mixed 
effects in other initiatives depending on how they were measured. A clear 
definition of the initiative’s unit of change and of analysis is necessary to 
understand these outcomes appropriately. 
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Determining Units of Change and Analysis 

The transformation of the educational systems is, by design, the ultimate 
intended outcome ‘of systemic educational reform. From this perspective, the 
unit of change of these reforms should be defined as the entire educational 
system regardless of whether this system is a state, city, or district. However, 
several major issues which are unique to systemic educational reforms pose 
significant dilemmas to systemic reformers in choosing the entire system as the 
unit of change of these reforms. These issues are: (1) the magnitude and 
complexity of the proposed task; (2) the unavailability of resources 
commensurate with the task; and (3) the risks involved in intervening in the 
system in its entirety without first pilot testing interventions at a smaller scale 
(Gomez, 1997). Therefore, reformers need to design strategies to have an 
impact on the entire system without working with the entire system at once. At 
the same time, evaluators need to work with reformers to: (1) identify a unit of 
change that will facilitate the transformation of the system while starting small 
and scaling-up; (2) measure such a major multi-dimensional transformation; and 
(3) select a unit of analysis within that unit of change. 

The identification of a systemic reform’s outcomes will depend mainly on the 
overall goals and objectives of the initiative. The unit of change will be the focus 
of the reform and, consequently, of its evaluation. In addition, outcomes obtained 
at the unit of change will be interpreted differently depending on the unit of 
analysis that has been selected to answer a particular evaluation question. The 
following section presents some examples of outcomes of systemic educational 
reform of interest to evaluators. 

Key Outcomes in Systemic Educationai Reform 

Outcomes of systemic educational reforms can be organized in terms of whether 
they are the result of top-down or bottom-up approaches. Top-down approaches 
entail changes at the legislative and policy levels while bottom-up approaches 
refer to changes occurring at the grassroots level such as schools. 

Educational policy is a top-down outcome of interest to evaluators because it is 
highly interrelated with other aspects of the reforms. Educational policy is 
necessary but not sufficient to achieve true systemic change because 
“connections between policy alignment and school change are the black box of 
systemic reform” (Zucker & Shields, 1998; p.8). Evaluators working with changes 
in educational policy may need to differentiate which policies can be attributed 
directly to the reform and which ones can be windows of opportunity to enable 
the implementation of the reform. Some examples of educational policy of 
interest to evaluators are: (1) shared vision of quality Science and Mathematics 
education at the state or district levels; (2) standards, curricula, and strategic 
planning for quality Science and Mathematics education at the state or district 
levels; (3) laws, regulations, and monitoring procedures for special populations in 
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the areas of equity, fairness, and accommodations at the state or district levels; 
and (4) administration of Federal programs such as Title I and Title II consistent 
with quality Mathematics and Science programs at the state and district levels. 

Changes taking place at the school level are examples of bottom-up changes. 
When initiatives are being truly systemic, the transformations that they pursue 
surpass the limits of the classroom and encompass the school as a whole. The 
sustainability of the reform is more likely to be supported when the culture of a 
school is changed because the participants will be less likely to return to previous 
patterns of behavior and communication. Since the sociological literature defines 
culture as the attitudes, beliefs, values, expectations, knowledge, opportunities, 
and materials of a group (Chinoy & Hewitt 1975; Federico, 1975; Remmling & 
Campbell, 1970), an evaluator could expect to find changes in elements of this 
definition which could be considered outcomes of systemic educational reform at 
the school level. 

Evaluators may choose to obtain information on variables such as the ones 
presented below as baseline measures against which to compare changes after 
the systemic educational reform has been implemented. These variables are 
discussed in further detail in the book about the evaluation of systemic 
educational reforms from which this paper was adapted. 

1. Percent of qualified staff teaching Science and Mathematics. 

2. Degree of shared beliefs and logic of action for teaching Science and 
Mathematics among the teaching staff. 

3. Availability of quality curricula and review processes for Mathematics and 
Science. 

4. Use of standards-based teaching and learning and assessment strategies in 
Science and Mathematics. 

5. Existence of communities of learners among Science and Mathematics 
teaching staff and other members of the school communities. 

6. Hours of standards-based professional development experiences for 
Science and Mathematics teaching staff. 

7. Level of empowerment (i.e., autonomy, accountability, decision-making 
opportunities, collegiality and collaboration, mastery of content, self- 
assessment, and institutional support among other elements) of teachers of 
Science and Mathematics. 

8. Expenditures per student for Science and Mathematics. 

9. Availability of materials and equipment for Science and Mathematics 
instruction. 

10. Level of participation of parents and community in Mathematics and Science 
education. 

11. Level of academic entrepreneurship (i.e., school-initiated outreach efforts to 
obtain additional human and fiscal resources to support and sustain the 
reform) in building and maintaining quality Mathematics and Science 
instruction. 
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12. Existence of partnerships with community organizations to leverage 
resources and strengthen the Mathematics and Science reform. 

Positive student outcomes are “the main reason for being” of student-centered 
systemic educational reforms. The central focus of most systemic educational 
reforms is the achievement of challenging academic standards by all students 
that can be demonstrated through improvements in student academic 
achievement. Student academic achievement is interrelated with aspects of the 
initiatives such as their visions of quality education, expectations of performance 
for participants, definitions of equity, and designs of professional development 
interventions among others. Further, student academic achievement is a 
concrete indicator of progress that is associated with other areas of student 
success such as college and job placement. Thus, evaluators of systemic 
educational reforms are often expected to provide evidence of having an impact 
on student academic achievement as an indicator of the value added by the 
reforms choosing an appropriate data collection and reporting design that meets 
the needs of the initiatives and of their multiple stakeholders. 

Within the context of systemic educational reforms, student outcomes extend 
beyond student academic achievement. Participant variables such as course 
enrollment, college placement, and job placement as well as involvement and 
quality of performance in activities such as Science Fairs and Mathematics 
Olympics can be used to examine trends or changes in student achievement that 
reflect student academic achievement beyond test scores and course grades. 
Further, evaluators of systemic educational reforms may be interested in looking 
at variables such as attitudes towards Science and Mathematics and student 
empowerment as indicators of the reform’s impact. The following section 
presents some plausible alternatives to measure these variables within 
evaluations of systemic educational reforms. 

Some Methodological Alternatives in Systemic Educational Reform 
Evaluation 

Because of the complexity and dynamic nature of systemic educational reforms, 
a research approach is proposed as an alternative to measure and document 
outcomes. The scarcity of literature available on evaluation of systemic 
educational reforms could motivate evaluators to contribute to the development 
of an emerging field of research particularly since the visions of what evaluation 
is, what evaluation should be, and what evaluation will be are constantly 
changing. A research approach to evaluation could also provide enough flexibility 
to evaluators and reformers to reformulate the evaluation questions and redesign 
the evaluation as the information needs of the program evolve. 

A research approach to evaluation will be more methodologically sound if the 
variables of interest are measured in multiple ways. By comparing findings using 
multiple qualitative and quantitative data collection strategies as suggested in the 
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literature (Laguarda, Goldstein, Adelman, & Zucker, 1998), evaluators can 
identify trends and lead reformers to make pertinent mid-course corrections 
within the reforms. Combinations of qualitative and quantitative data collection 
strategies (Mark, Feller, & Button, 1997; Hedrick, 1994; Reichardt & Rallis, 1994; 
Yin, 1994) to look at student achievement and the use of internal and external 
tests to look at variables such as student academic performance are examples of 
these multiple approaches discussed elsewhere (see Davila & Gomez, 
1994:1995; Davila, Gomez & Vega, 1996; and Davila, 1999 among others for 
specific examples). 

The diverse dimensions involved in the evaluation of systemic educational 
reforms may lead the evaluator to envision the evaluation as a series of 
coordinated studies where each one is designed to answer specific questions. 
Each one of these studies could provide valuable insights into some aspect of 
the educational reform while all of them together could provide those insights into 
the overall reform and its effects. Such studies could be conducted on sub- 
systems, reform issues, or local contexts as they are discussed in the book that 
serves as the foundation of this paper. Each one of these studies will present its 
unique challenges to evaluators who may need to start thinking about to structure 
them in a way that the information that they yield is meaningful. 

Final Comments: Using Outcomes to Drive Systemic Educational Reform 

The identification and definition of outcomes of systemic educational reforms 
clearly reflects what is valued by the reform at a particular point in time. By 
selecting what will be measured and how it will be measured, evaluators and 
reformers are explicitly defining what is important for tbe reform because we 
usually measure what we value. Thus, by defining outcomes and measuring 
those outcomes, evaluators and reformers will be driving the reforms. 
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